Skip to content

test: guard same-job compile ratios on CPU time with wall backstops - #1162

Merged
kvlonge merged 2 commits into
mainfrom
test/cpu-ratio-guards
Oct 19, 2026
Merged

kvlonge merged 2 commits into
mainfrom
test/cpu-ratio-guards

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Why

Two compile performance guards compared two wall times measured in the same job and failed at random on shared runners by hairlines (plan vs warm compile 2.60 against a 2.6 limit; cache-write wall overhead 1.21 against 1.20). Wall time on shared runners is noisy; CPU time (user+sys) is steady and still catches added work.

Changes

  • Plan vs warm compile and cache-write overhead now assert a CPU-time ratio as the tight limit, plus a looser wall-time backstop for lost parallelism, lock waits and I/O.
  • The plan guard and every fresh-process cache path now log CPU time for both sides, so later calibration uses exact CPU ratios.
  • Limits come from 31 jobs per guard over the last two days (all branches, including release PRs from main). Each CPU limit is at least 1.15x the worst recent CPU ratio.
Guard Old New Worst recent
Plan/compile CPU none 3.1 2.69 (estimated)
Plan/compile wall 2.6 3.3 2.60
Cache-write CPU (3k/5k) 1.10 1.25 1.089
Cache-write CPU (10k) 1.15 1.25 1.061
Cache-write wall 1.20 1.35 1.178

The plan CPU ratio was not logged before, so its worst value is estimated from the planning-scaling test in the same job.

The warm, edit and project-rewrite to cold wall ratios (0.70/0.70/0.80, worst 0.61) keep their wall limits: CPU per path was not logged before, so they have no CPU data to calibrate from yet.

Verification

  • Collection of both changed test modules.
  • ruff, ty, fensu and make check-ci passed.
  • Timing guards run in CI.

The plan-vs-compile and cache-write overhead guards compared two wall
times in one job and failed at random on shared runners by hairlines.
CPU time (user+sys) is now the tight limit and a looser wall-time ratio
remains as a backstop for lost parallelism, lock waits and I/O.

Limits are calibrated from recent CI job logs: each CPU limit clears the
worst recent CPU ratio by at least 15%. The plan guard and the per-path
cache measurements now log CPU time for both sides.
@kvlonge
kvlonge enabled auto-merge (squash) October 19, 2026 16:59
@kvlonge
kvlonge merged commit 2c22eb5 into main Oct 19, 2026
68 of 72 checks passed
@kvlonge
kvlonge deleted the test/cpu-ratio-guards branch October 19, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant